Skip to content

feat: OpenRouter video surface drift leg (cost-safe live models canary) - #323

Merged
jpr5 merged 2 commits into
mainfrom
feat/openrouter-video-drift-leg
Jul 22, 2026
Merged

feat: OpenRouter video surface drift leg (cost-safe live models canary)#323
jpr5 merged 2 commits into
mainfrom
feat/openrouter-video-drift-leg

Conversation

@jpr5

@jpr5 jpr5 commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

What this covers

The first net-new live drift surface beyond the original three (OpenAI /
Anthropic / Gemini). The OpenRouter async video-generation proxy
(src/openrouter-video.ts, video-proxy-shared.ts) is shipped on master but
had zero drift coverage. This adds the openrouter-video surface:

  • Surface registry — new openrouter-video slug in surface-registry.ts
    pointing at the OpenRouter video handlers, so the drift-report collector
    routes any drift block from this leg to the right source file (exit-2
    auto-fix lane) instead of quarantining it.
  • Live models canary (FREE)listOpenRouterVideoModels authenticates
    with OPENROUTER_API_KEY and hits the public listing endpoint
    GET /api/v1/videos/models, asserting the video-model families aimock's
    proxy mirrors (seedance, sora — the DEFAULT_OPENROUTER_VIDEO_MODELS
    set) are still present. Gated with describe.skipIf(!OPENROUTER_API_KEY).
  • Static envelope-shape legs (no key, run in CI) — drive the aimock server
    over HTTP and triangulate the submit response, completed status-poll, and
    models-listing envelopes the proxy emits against hand-authored exemplars.
    These exercise the real handler + triangulate + collector routing path,
    not a unit fake.
  • Workflow wiringOPENROUTER_API_KEY added to the drift-job env: in
    both .github/workflows/test-drift.yml and .github/workflows/fix-drift.yml.

Cost-safe scoping rationale (no paid generation)

Video generation costs real money per job, so nothing here submits a
generation
. The daily live-reality check is the models listing endpoint
(free, metadata only). Job-lifecycle envelope shapes are asserted statically
against the mock's own output — no live paid call. A live job would be the only
way to triangulate the content download against reality; that is deliberately
deferred as a follow-up rather than incurring generation cost. Grok Imagine
(xAI) and Veo (Google) live canaries are also deferred — their listing
endpoints live on different hosts and need their own keys.

Red-green proof

The real failure surface is the drift collector / triangulation, not a unit
fake. Structural red-green was captured locally against pnpm run test:drift.

RED — the mock envelope was deliberately diverged by renaming
supported_resolutionsresolutions in handleOpenRouterVideoModels
(src/openrouter-video.ts), then running the leg:

 ❯ src/__tests__/drift/openrouter-video.drift.ts (4 tests | 1 failed | 1 skipped)
   ✓ submit returns { id, polling_url, status }
   ✓ completed status poll returns { id, status, unsigned_urls, usage }
   × models listing returns { data: [ { id, name, supported_* , ... } ] }
     →
API DRIFT DETECTED: OpenRouter video models listing
  Surface: openrouter-video

  1. [critical] LLMOCK DRIFT — field in SDK + real API but missing from mock
     Path:    data[].supported_resolutions
     SDK:     array<string>
     Real:    array<string>
     Mock:    <absent>

  2. [info] MOCK EXTRA FIELD — in mock but not in real API
     Path:    data[].resolutions
     Mock:    array<string>

 Test Files  1 failed (1)
      Tests  1 failed | 2 passed | 1 skipped (4)

The failure carries the Surface: openrouter-video marker, so the collector
routes it structurally — proving the whole triangulate → formatDriftReport →
collector path is exercised.

GREEN — reverted the one-line divergence, re-ran:

 ✓ src/__tests__/drift/openrouter-video.drift.ts (4 tests | 1 skipped) 14ms

 Test Files  1 passed (1)
      Tests  3 passed | 1 skipped (4)

Full suite also green: pnpm run test:drift → 16 passed | 9 skipped files;
drift-collector.test.ts (registry coverage/integrity) → 160 passed; eslint,
prettier --check, and tsdown build all clean.

Live validation dependency (deferred to CI)

The live canary's red-green is deferred to CI: OPENROUTER_API_KEY is not
available in the local agent environment, so faking it was not an option. Once
the secret is mirrored to repo secrets, the drift-live-pr job (and the daily
drift job) will run the live leg. Until then the live canary
describe.skipIf-skips in CI — that is expected, not a failure.
This has not
been claimed live-verified; only the structural red-green was run locally.

Notes

  • src/__tests__/drift/* + .github/workflows/* only — not in the published
    npm package, so no version bump / CHANGELOG.

🤖 Generated with Claude Code

Adds the first net-new live drift surface beyond the original three
(OpenAI/Anthropic/Gemini). The OpenRouter async video-generation proxy
(src/openrouter-video.ts, video-proxy-shared.ts) shipped on master with
zero drift coverage; this closes that gap without incurring any paid
video generation.

- New surface slug `openrouter-video` in surface-registry.ts pointing at
  the OpenRouter video handlers, so the collector routes its drift blocks.
- Live canary (FREE, metadata only): listOpenRouterVideoModels hits
  GET /api/v1/videos/models and asserts the model families aimock mirrors
  (seedance, sora) are present. Gated on OPENROUTER_API_KEY.
- Static envelope-shape legs (no key, run in CI): triangulate the submit,
  completed-status, and models-listing envelopes the proxy emits against
  hand-authored exemplars — exercising the real handler + collector path.
- Wire OPENROUTER_API_KEY into the drift job env in test-drift.yml and
  fix-drift.yml so the live canary runs once the secret is mirrored.
@pkg-pr-new

pkg-pr-new Bot commented Jul 22, 2026

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/@copilotkit/aimock@323

commit: f75c8a3

@jpr5
jpr5 marked this pull request as ready for review July 22, 2026 23:37
@jpr5
jpr5 merged commit edb5e9a into main Jul 22, 2026
29 checks passed
@jpr5
jpr5 deleted the feat/openrouter-video-drift-leg branch July 22, 2026 23:38
jpr5 added a commit that referenced this pull request Jul 23, 2026
…ed) (#327)

## What

Adds a **live** queue-lifecycle drift probe for the fal.ai queue surface
(`src/fal.ts`). Rows 25/26 of the drift-coverage audit were STATIC-only;
this closes the live gap for the queue surface with a **cost-safe**
canary that self-activates once `FAL_KEY` is mirrored to repo secrets.

Mirrors the OpenRouter-video precedent (#323): live leg is
`describe.skipIf(!FAL_KEY)`, so it **skips in CI until the secret
lands**, then runs automatically.

## Changes

- **`src/__tests__/drift/providers.ts`** — new
`falQueueLifecycleCanary(apiKey, modelId, input)`: drives the REAL fal
queue API through **submit → status → immediate cancel**, returning the
three envelope bodies. Cancel is issued even if the status poll throws
(never leaves a job to RUN).
- **`src/__tests__/drift/fal-queue.drift.ts`** — new
`describe.skipIf(!FAL_KEY)("fal.ai queue lifecycle (live, cost-safe)")`
block: asserts the load-bearing submit/status/cancel fields on the real
envelope and triangulates each step (exemplar × real × aimock mock)
under the `fal-queue` surface slug. Completed-**result** envelope stays
STATIC-only (never fetched — that is the only paid retrieval).
- **`.github/workflows/test-drift.yml` + `fix-drift.yml`** — add
`FAL_KEY: ${{ secrets.FAL_KEY }}` to every drift-job env block,
alongside the existing provider keys (minimal/additive).

Surface registry already contains the `fal-queue` slug — no registry
edit needed.

## Cost note

fal bills **compute only when a queued job RUNS**. The canary submits
(free) and cancels immediately while the job is still `IN_QUEUE`, so
**expected cost is \$0**. It never fetches the result endpoint. Cheapest
reliably-available model (`fal-ai/flux/schnell`) bounds the worst case;
**residual exposure is at most one sub-cent generation** in the rare
race where the model completes before the cancel lands.

## Red-green

Live red-green cannot run locally (`FAL_KEY` is not yet a repo secret
and 1Password access is guard-blocked) — **CI will capture the live
red-green once `FAL_KEY` is mirrored.** Structural red-green proven
locally against the real drift machinery:

**RED** — forced the live block to run with a deterministically drifted
real submit envelope (`queue_position` as string vs the mock's number):
```
API DRIFT DETECTED: fal.ai queue submit (live)
  Surface: fal-queue

  1. [critical] TYPE MISMATCH between real API and mock: string vs number
     Path:    queue_position
     SDK:     number
     Real:    string
     Mock:    number
Test Files  1 failed (1)
```

**GREEN** — reverted; live block skips (no `FAL_KEY`), static legs pass:
```
✓ src/__tests__/drift/fal-queue.drift.ts (5 tests | 1 skipped)
Test Files  1 passed (1)
```

Full suite: `Test Files 16 passed | 9 skipped (25) Tests 93 passed | 65
skipped`. Typecheck (touched files), prettier, eslint, build all clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant